(12) INTERNATIONAL APPLICATION PUBLISHED UNDER THE PATENT COOPERATION TREATY (PCT) 



(19) World Intellectual Property Organization 
International Bureau 

(43) International Publication Date 
26 July 2001 (26.07.2001) 




PCT 



(10) International Publication Number 

WO 01/54067 A2 



(51) International Patent Classification 7 : G06T 5/00, 1/60 

(21) International Application Number: PCT/USO 1/02 180 

(22) International Filing Date: 22 January 2001 (22.01.2001) 

(25) Filing Language: English 

(26) Publication Language: English 



(30) Priority Data: 

60/177,575 



22 January 2000 (22.0 1 .2000) US 



(71) Applicant: KAIROS SCIENTIFIC INC. [US/US]; 
Building 62, 3350 Scott Boulevard, Santa Clara, CA 95054 
(US). 

(72) Inventors: YANG, Mary, IVL; 176 Covington Court, 
San Jose, CA 95136 (US). BYLINA, Edward, J.; 2629 
Bon Bon Drive, San Jose, CA 95148 (US). COLEMAN, 
William, X; Apartment 10, 750 R Shoreline Boulevard, 
Mountain View, CA 94043 (US). DILWORTH, Michael, 
R.; 220 32nd Avenue, Santa Cruz, CA 95062 (US). ROB- 
LES, Steven, J.; 165 E. Reed Street, Apt. 4, San Jose, 



CA 951 12 (US). YOUVAN, Douglas, C; 176 Covington 
Court, San Jose, CA 95136 (US). 

(74) Agents: SHUSTER, Michael, J. et al.; McCutchen. 
Doyle, Brown & Enersen, LLP, Three Embarcadero 
Center, San Francisco, CA 941 11-4067 (US). 

(81) Designated States (national): AE, AG, AL, AM, AT, AU, 
AZ, BA, BB, BG, BR, BY, BZ, CA, CH, CN, CR, CU, CZ, 
DE, DK, DM, DZ, EE, ES 5 FI, GB, GD, GE, GH, GM, HR, 
HU, ID, IL, IN, IS, JP, KE, KG, KP, KR, KZ, LC, LK, LR, 
LS, LT. LU, LV, MA, MD, MG, MK, MN, MW, MX, MZ. 
NO, NZ, PL, PT, RO, RU, SD, SE, SG, SI, SK, SL, TJ, TM, 
TR, TT, TZ, UA, UG, UZ, VN, YU, ZA, ZW. 

(84) Designated States (regional): ARIPO patent (GH, GM, 
KE, LS, MW, MZ, SD, SL, SZ, TZ, UG, ZW), Eurasian 
patent (AM, AZ, BY, KG, KZ, MD, RU, TJ, TM), European 
patent (AT, BE, CH, CY, DE, DK, ES, FI, FR, GB, GR, rE, 
IT, LU, MC, NL, PT, SE, TR), OAP1 patent (BF, BJ, CF, 
CG, CI, CM, GA, GN, GW, ML, MR, NE, SN, TD, TG). 

Published: 

— without international search report and to be republished 
upon receipt of that report 

[Continued on next page] 



(54) Title: VISUALIZATION AND PROCESSING OF MULTIDIMENSIONAL DATA USING PREFILTERING AND SORTING 
CRITERIA 




< 

o 

TT 
IT) 



o 



(57) Abstract: Complex multidimensional datasets generated by digital imaging spectroscopy can be organized and analyzed by 
applying software computer-based methods comprising sorting algorithms. Combinati these algorithms to images and graphical 
data, allow pixels or features to be rapidly and efficiently classified into meaningful groups according to defined criteria. Multiple 
rounds of pixel or feature select be performed based on independent sorting criteria. In one embodement sorting by spectral criteria 
(e.g., intensity at a given wavelength) is combined with sorting by temporal criteria (e.g., absorbance at a given time) to identify 
microcolonies of recombinant organisms harboring mutated genes encoding enzymes having desirable kinetic attributes and substrate 
specificity. Restriction of the set of pixels analyzed in a subsequent sort based on criteria applied in an earlier sort ("sort and lock" 
analyses) minimize computational and storage resources. User-defined criteria can also be incorporated into the sorting process by 
means of a graphical user interface that comprises a visualization tools including a contour plot, a sorting bar and a grouping bar, an 
image window, and a plot window that allow run-time interactive identification of pixels or features meeting one or more criteria, 
and display oftheir associated spectral or kinetic data? these methods are useful for extracting information from imaging data in 
applications ranging from biology and medicine to remote sensing. 
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VISUALIZATION AND PROCESSING OF MULTIDIMENSIONAL DATA USING PREETLTERING AND SORTING 

CRITERIA 

RELATED APPLICATION DATA 

This application claims the benefit of U.S. Provisional Application No. 
60/177,575, filed January 22, 2000 and U.S. Provisional Application No. 60/186,034, 
filed March 1, 2000, the entire disclosures of which are hereby incorporated by 
5 reference in their entirety. 

FIELD OF THE INVENTION 

The current invention relates generally to the visualization and processing of 
multidimensional data, and in particular, to data formed from a series of images. 
BACKGROUND OF THE INVENTION 
1 0 Sophisticated analysis of imaging data requires software that can rapidly 

identify meaningful regions of the image. Depending on the size and number of 
regions, this process may require evaluating very large datasets, and thus efficient 
sorting of the data is essential for finding the desirable elements. In the present 
invention, regions of interest (ROIs) in previous feature-based imaging spectroscopy 
1 5 are extended to include pixel-based analyses. This requires new algorithms, since the 
size of a pixel-based analysis can be more than 1 000 times larger than that of a 
feature-based analysis. In addition to requiring a burdensome amount of processing 
time, prior art sorting algorithms that may have been adequate to categorize and 
classify relatively noiseless feature data are not necessarily successful in sorting 
20 single-pixel spectra without additional parameters or human intervention. 

In cases in which human intervention is advantageous, the present invention 
includes a means for combining machine and human intelligence to enhance image 
analysis. For example, the present invention provides a method for combining sorting 
by spectral criteria (e.g., intensity at a given wavelength) and sorting by temporal 
25 criteria (e.g., absorbance at a given time). Sorting enables the user to classify large 
amounts of data into meaningful and manageable groups according to defined criteria. 
The present invention also allows for multiple rounds of pixel or feature selection 
based on independent sorting criteria. Methods are presented for extracting useful 
information by combining the analyses of multiple datasets and datatypes (e.g., 
30 absorbance, fluorescence, or time), such as those obtained using the instruments and 
methods disclosed in U.S. Patent Nos. 5,859,700 and 5,914,245, and in U.S. Patent 
App.No. 09/092,316. 
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The methods described herein are useful for a number of applications in 
biology, chemistry and medicine. Biomedical applications include high-throughput 
screening (e.g., pharmaceutical screening) and medical imaging and diagnostics (e.g., 
oximetry or retinal examination). Biological targets include live or dead biological 
5 cells (e.g., bacterial colonies or tissue samples), as well as cell extracts, DNA or 
protein samples, and the like. Sample formats for presenting the targets include 
microplates and other miniaturized assay plates, membranes, electrophoresis gels, 
microarrays, macroarrays, capillaries, beads and particles, gel microdroplets, 
microfluidic chips and other microchips, and compact discs. More generally, the 

1 0 methods of the present invention can be used for analysis of polymers, optical 
materials, electronic components, thin films, coatings, combinatorial chemical 
libraries, paper, food, packaging, textiles, water quality, mineralogy, printing and 
lithography, artwork, documents, remote sensing data, computer graphics and 
databases, or any other endeavor or field of study that generates multidimensional 

1 5 data. 

SUMMARY OF THE INVENTION 

The present invention provides methods, systems and computer programs for 
analyzing and visualizing multidimensional data. Typically, the first two dimensions are 
spatial and the third dimension is either spectral or temporal. (Although the term spectra 

20 or kinetics may be used herein, the methods described are of general applicability to both 
forms of vector data.) The invention includes a graphical user interface and method that 
allows for the analyses of multiple data types. For example, datastacks of fluorescence 
emission intensity, absorbance, reflectance and kinetics (changes in signal over time) can 
be analyzed either independently or on the same sample for the same field of view. 

25 Fluorescence measurements involving fluorescence resonance energy transfer (FRET) 
can also be analyzed. A key feature of the present invention is that data analysis can be 
performed in series. Thus, for example, the results of sorting pixels or features within one 
image stack can be applied to subsequent sorts within image stacks. The present invention 
also includes methods to prefilter data. Thus, for example, pixel-based analysis can be 

30 performed, wherein features are selected based on particular criteria and a subsequent sort 
is restricted to pixels that lie within the selected features. These sorting methods are 
guided by the heuristics of parameters input by the user. This is especially beneficial 
when expert knowledge is available. Thus, for example, the user can select a particular 
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spectrum with desirable characteristics (a target spectrum) from a spectral stack, and the 
program will automatically classify all of the spectra obtained from the image stack by 
comparing each of the unclassified spectra to the target spectrum, calculating a distance 
measure, and sorting the spectra based on their distance measure. The classified (sorted) 
5 spectra are then displayed in the contour plot window or other plot windows. 

Sorting can also be used for sequentially analyzing images and graphical data, 
such that the pixels that are ultimately displayed are restricted by at least two 
independent criteria. For example, pixels or features that have been extracted based 
on selected spectral criteria (e.g., absorbance) can be further analyzed based on 
10 temporal criteria (e.g., kinetics). This method of combined analysis provides a means 
for rapidly and efficiently extracting useful information from massive amounts of 
data. A further embodiment of sequential sorting involves discarding unwanted data 
during the sorting process. This 'sort and lock' procedure provides a useful new tool 
for data compression. This method for sorting and displaying multidimensional data 
1 5 from an image stack comprises the steps of: (a) selecting a subset of pixels from an 
image by a first algorithm; (b) discarding the pixels that are not selected; (c) selecting 
a subset of the remaining pixels by a second sorting algorithm; and (d) automatically 
indicating the final selection of pixels by back-coloring the corresponding pixels in 
the image. This type of multidimensional analysis can also be performed by 
20 manipulating the contour plot window. The method comprises the steps of (a) sorting 
the pixels by a first algorithm; (b) automatically indicating on the contour plot pixels 
sorted by the first algorithm; (c) selecting a subset of pixels in the contour plot; (d) 
sorting the subset of pixels by applying a second algorithm; (e) selecting a reduced 
subset of pixels in the contour plot; and (f) automatically indicating the final selection 
25 of pixels by backcoloring the reduced subset of pixels in the image. The present 
invention also provides a method for displaying a grouping bar that can be used to 
analyze images and graphical data within the graphical user interface ("GUI")- The 
grouping bar enables the user to segregate groups of pixels or features within a 
contour plot, and thereby facilitates independent sorting and backcoloring of the 
30 individual groups of pixels or features in the image. The methods of the present 
invention are applicable to a variety of problems involving complex, 
multidimensional, or gigapixel imaging tasks, including (for example) automated 
screening of genetic libraries expressing enzyme variants. 
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According to one embodiment of the invention, a method for analyzing digital 
image data is provided, said method comprising (a) loading into a computer memory a 
plurality of data stacks wherein each data stack comprises pixel intensity data for a 
plurality of images, the pixel intensity data expressed as a function of: (i) pixel 
5 position, (ii) a first non-positional variable, and (hi) a second non-positional variable, 
wherein within a data stack, the value of the first non-positional variable is not 
constant and the value of the second non-positional variable is constant, and wherein 
between data stacks, the value of the second non-positional variable differs; (b) 
generating for a plurality of pixels within a first data stack, a plurality of first 
1 0 functions that relate pixel intensity to the first non-positional variable; (c) sorting the 
pixels within the first stack according to a first value obtained by applying a 
mathematical operation to the first functions generated for the plurality of pixels; (d) 
selecting a first set of sorted pixels; (e) generating for a plurality of pixels within the 
first set, a plurality of second functions that relate pixel intensity to the second non- 
15 positional variable; and (f) sorting the pixels within the first set according to a second 
value obtained by applying a second mathematical operation to the second functions 
generated for the plurality of pixels within the first set. The non-positional variables 
may be selected from a wide range of different parameter types that indicate, e.g., the 
time the data were captured, or ,e.g., a condition such as wavelength, temperature, pH, 
20 chemical activity (such as, e.g., the concentration of an enzyme substrate or enzyme 
inhibitor, or the concentration of a drug or other chemical component), pressure, 
partial pressure of a gaseous chemical, or ionic strength, etc. under which the data 
were captured. 

According to another embodiment, the invention provides a graphical user 
25 interface ("GUT') for display and analysis of digital image data comprising (a) a 

reference window for displaying a reference image comprising pixels; (b) a contour 
plot window for indicating pixel location along a first dimension, indicating a non- 
positional variable (such as, e.g., time, wavelength, temperature, pH, chemical 
activity, pressure, partial pressure of a gaseous chemical, or ionic strength, etc.) along 
30 a second dimension, and indicating pixel intensity by a variable signal appearing 
along the second dimension, said contour plot window further comprising (i) a 
grouping bar for grouping together pixels for analysis; and (ii) a selection bar for 
selecting pixels that are thereby indicated in the reference window and plotted in the 
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plot window; (c) a plot window for displaying a plot of pixel intensity as a function of 
the non-positional variable. 

BRIEF DESCRIPTION OF THE DRAWINGS 
The file of this patent contains at least one drawing executed in color, copies 

5 of this patent with color drawing(s) will be provided by the Patent and Trademark 
Office upon request and payment of the necessary fee. 

FIG. 1 illustrates the graphical user interface. This GUI utilizes coordinately- 
controlled windows that are interactive in run-time. Note that in subsequent figures 
there are two vertical color bars and one horizontal color bar. The horizontal bar 

1 0 under the contour plot provides a color-coded scale for the variable (e.g. spectra or 
kinetics) plotted within the contour plot. There are two vertical color bars to the left 
of the contour plot. These two bars are separated by a vertical black line. The first 
vertical bar immediately to the left of the contour plot is a selection/mapping bar that 
is used to indicate rows that have been mapped or back-colored onto the image. The 

1 5 second vertical bar is a grouping bar that is used to delineate or segregate rows into 
user-defined groups of spectra or kinetics data. By default, the entire contour plot is 
equivalent to one group and in this case, the grouping bar is not shown. Likewise, if 
nothing had been selected or mapped onto the image, the selection/mapping bar 
would not be seen. 

20 FIG. 2 shows a workspace window demonstrating the layout of multiple 

projects containing absorption, fluorescence, and kinetic data. The individual 
timepoint images are shown for the kinetics Image Stack. In a spectral datastack (not 
shown), the individual timepoint images are replaced by images taken under specific 
wavelength conditions. 

25 FIG. 3 shows a graphical representation of four dimensional data, including 2 

spatial dimensions (x andy), a temporal dimension (t), and a spectral parameter (e.g., 
wavelength, lambda). 

FIG. 4 shows a user interface which demonstrates the ability of the software to 

display multiple windows. 
30 FIG. 5 shows why picking a 'positive 1 (pixel or feature) is best done by pixel- 

based rather than by feature-based analysis. For example, in complex images such as 
confluent groups of microcolonies (Panel A\ feature extraction (Panel D) is inferior to 
single-pixel analysis (Panel E) for identification of the 'fastest' colonies (represented 
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in black) because of edge effects and other artifacts. This effect becomes more 
apparent as the features within a target become smaller and approach the apparent 
pixel size, i.e., each feature in Panel B covers only a few pixels and appears as a 
single 'blob 1 (Panel Q if it is processed based on feature extraction. 
5 FIG. 6 shows a screen capture of a GUI, demonstrating reflectance imaging 

spectroscopy and single-pixel analyses of M&M candies as test targets. The' four 
windows in this image correspond to those depicted in FIG. 1. The vertical mapping 
bar is depicted by the yellow and green tick mark on the left side of the contour. 
Since the entire contour is a single group, the vertical group selection bar is not 

1 0 visible. Otherwise, it would appear in a similar manner to the right of the mapping 
bar; on the other side of the vertical black line. Two single-pixel spectra have been 
highlighted for microscopic regions within a yellow and a green candy. 

FIG. 7 shows how contour plots can be sorted to group single-pixel spectra by 
similarity or other criteria. Unsorted data (Panel A) can be processed to group spectra 

1 5 with similar attributes (Panel E). User-input heuristics is important in guiding the 
particular criteria (i.e., a series of algorithms) that are required to sort a dataset to a 
level of refinement appropriate for back-painting the image window. Additional 
sorting algorithms that are not shown are "Sort by channel of maximum value" and 
"Sort by ratio of two channels". Sorts can be performed on either variable, fall-scale, 

20 or derivative data, or a combination of these. 

FIG. 8 shows a contour map utilizing vertical bars for grouping and mapping 
pseudocolors to the image window at single-pixel resolution. 

FIG. 9 shows an example of backcoloring an image of M&M candies based on 
spectral sorting of single-pixel data. Two vertical bars are shown. A bar immediately 

25 to the left of the contour plot is used for selection and mapping back onto the image. 
It uses pseudocolors approximating the real color of the candies. This color code is 
maintained in the plot window for each of the seven different colored candies, along 
with an eighth plot for the black background. Average spectra, each of which 
corresponds to the selected rows from the eight different groups, are plotted. All 

30 spectra are shown with full-scale (variable) deflection. Note that the letter C M' can be 
distinguished in yellow M&M's. This is a result of individually sorting the yellow 
category spectra by the maximum intensity while in a fixed scale mode. As seen by 
the yellow selection bar to the left of the contour, only the upper portion of the yellow 
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group is selected. The second vertical bar (left of the vertical black line) is the 
grouping bar used to delineate or segregate rows into user-defined groups of spectra 
or kinetics data. 

FIG. 10 shows a flowchart for creating an Absorbance analysis. 
5 FIG. 1 1 shows a kinetics analysis of E. coli microcolonies expressing genetic 

variants of the enzyme Abg (Agrobacterium beta-glucosidase) on an assay disk. 
Formation of red indigo due to hydrolysis of the chromogenic substrate Red-gal 
indicates galactosidase activity in the microcolony. Formation of blue indigo due to 
hydrolysis of the chromogenic substrate X-glu indicates glucosidase activity in the 

1 0 microcolony. The data were recorded by an RGB camera, i.e., a linear camera or 
flatbed scanner such as a Hewlett Packard ScanJet 6200C . A 2-minute image has 
been subtracted from a 24-minute image, and the operation of selecting the top 5% of 
the reddest and bluest colonies (i.e., Red-gal and X-glu cleavage) is about to be 
applied, as shown in FIG. 12. This dialog box provides for other operations involving 

1 5 either AND or OR operations on the values of pixels in the red, green, and blue 
channels. 

FIG. 12 shows the operation of a dialog box wherein pixels have been selected 
whose values in the red and blue channels are at or above the top 5% of the pixel 
value range in the image. These selected pixels are shown in magenta since the AND 

20 option is checked. Only these pixels and their close neighbors will be further 

analyzed, thus greatly reducing the amount of computation and data storage required. 

FIG. 13 shows a generalized flowchart of steps which can be used to identify 
enzyme variants exhibiting both the fastest kinetics and highest substrate specificity. 
FIG. 14 shows an example of an Abg REM-bluest pixel sort displayed in the 

25 Graphical User Interface of the Kcat instrument. This experiment involves spectral 
analysis to assay the results of creating an Abg library using recursive ensemble 
mutagenesis ("REM"). Random mutagenesis creates a range of genetic variants with 
potentially different substrate specificities. The chromogenic substrates employed in 
the assay are Red-gal and X-glu. A grouping bar (red) marks the pixels with the 

30 lowest 540 nm : 610 nm absorption ratio (the bluest pixels). The selection/mapping 
bar (blue) indicates some of these bluest pixels. The software back-colors these 
selected pixels on the image in the GUI and draws out the spectral traces of each of 
the selected pixels in the plot window. Sorting was based on the 540 nm : 610 nm 
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ratio. Pixels within microcolonies that favor the substrate Red-gal are sorted to the 
top of the contour plot and pixels within microcolonies that favor the substrate X-glu 
are sorted to the bottom of the contour plot. 

FIG. 15 shows an Abg REM - 'Fastest/Bluest* Pixel Sort. The Graphical User 
5 Interface of the Kcat instrument shows the kinetic analysis of an Abg mutagenesis 
experiment using substrates Red-gal and X-glu. All of the pixels which were 
analyzed in the 'Abg REM - Bluest Pixel Sort 5 analysis are analyzed in this 
timecourse analysis. The software determines the absorbance value for each selected 
pixel at every timepoint and displays the resulting kinetic traces in the contour plot. A 

10 grouping bar (red) marks the kinetic plots of the pixels with the greatest 610 nm : 540 
nm absorption ratio (the bluest pixels). After the 'bluest pixel' group was moved to 
the top of the contour plot, it was sorted by the maximum value at T = 600 seconds, a 
timepoint where the rate of change in intensity is linear. The selection/mapping bar 
(blue) indicates the pixels at the top of this result. These pixels correspond to the 

1 5 subset of 'bluest pixels' displaying the fastest product formation. The software 

backcolors these selected pixels onto the image in the GUI and draws out the kinetic 
traces for each of the selected pixels in the plot window. 

FIG. 16 shows an Abg REM - 'Fastest Pixel 5 Sort. Graphical User Interface 
of the Kcat instrument shows the kinetic analysis of an Abg mutagenesis experiment 

20 using the substrates Red-gal and X-glu. An image from the absorbance datastack is 
flat-fielded and the darkest pixels are selected for analysis. The software determines 
the absorbance value for each selected pixel at every timepoint and displays the 
resulting kinetics in the contour plot. Pixel data was sorted by 
the maximum value at T = 600 seconds, a timepoint where the rate of change in 

25 intensity is linear. A grouping bar (red) marks the kinetic plots of the pixels with the 
greatest absorbance at T = 600 (the 'fastest 5 pixels). The selection/mapping bar (blue) 
indicates some of these fastest pixels. The software backcolors these selected pixels 
on the image in the GUI and draws out the kinetic traces of each of the selected pixels 
in the plot window. 

30 FIG. 17 shows an Abg REM - 'Bluest /Fastest 5 Pixel Sort. The Graphical 

User Interface of the Kcat instrument shows the spectral analysis of an Abg 
mutagenesis experiment using the substrates Red-gal and X-glu. All of the pixels 
which were analyzed in the 'Abg REM - Fastest Pixel Sort 5 analysis are analyzed in 
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this spectral analysis. The software determines the absorbance spectrum for each 
selected pixel and displays the resulting spectra in the contour plot. A grouping bar 
(red) marks the spectral plots of the pixels with the greatest absorption at T = 600 
seconds (the fastest pixels). This group has been subsequently sorted by the ratio of 
5 intensities at 540 nm : 610 nm. The selection/mapping bar (blue) indicates the pixels 
at the top of this sort. These pixels correspond to the subset of 'fastest pixels' 
displaying the bluest product formation. The software backcolors these selected 
pixels on to the image in the GUI and draws out the spectral traces of each of the 
selected pixels in the plot window. 
1 0 FIG. 1 S shows a GUI displaying single-pixel absorption spectra from a slide 

of thin-sectioned tissue stained with hematoxylin and eosin (H&E). The image 
window shows a transverse section of cardiac tissue back-painted based on the sorted 
single-pixel spectra. Clear areas in the slide are sorted to the bottom of the contour 
plot and pseudocolored green. Pixels indicating hematoxylin absorption (610 nm) are 
1 5 painted blue and coincide with nuclei. Eosinophilic tissue is painted red. 

FIG. 19 shows desampled contour plots displaying single-pixel absorption 
spectra from a microscope slide stained with H&E. Panel A shows the initial unsorted 
contour plot in a variable scale mode. The plot window (A) shows single-pixel 
spectra corresponding to the colored tick marks in the contour plot (directly above). 
20 In Panel B, the variable scale spectra have been sorted by the ratio of optical 

absorbances at 540 nm compared to 610 nm. Thick blue and red lines in the plot 
window (B) correspond to spectral averages of pixels grouped above in the contour 
plot, and these are marked by blue and red bars, respectively. Panel C shows a 
derivative plot of these spectra with an eosin-specific inflection point at 530 nm. 
25 DETAILED DESCRIPTION OF THE INVENTION 

Unless otherwise defined, all technical and scientific terms used herein have 
the same meaning as commonly understood by one of ordinary skill in the art to 
which this invention belongs. Although methods and materials similar or equivalent . 
to those described herein can be used in the practice and testing of the present 
30 invention, suitable methods and materials are described below. 

All publications, patent applications, patents, and other references mentioned 
herein are incorporated by reference in their entirety. In case of conflict, the present 
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application, including definitions, will control. In addition, the materials, methods, 
and examples described herein are illustrative only and not intended to be limiting. 

Other features and advantages of the invention will be apparent from the 
following detailed description, the drawings, and from the claims. 
5 Layout of the Graphical User Interface 

The graphical user interface ("GUI") is diagrammed in FIG. 1. As shown, 
this GUI comprises four main interactive windows: Workspace, Image, Plot, and 
Contour. The workspace window has a look and feel similar to the Microsoft 
Windows Explorer, and it is used to organize and access projects and files. Typically, 

10 a project contains at least one stack of images (a datastack) taken under conditions in 
which a spectral or temporal variable is incremented by wavelength or time, 
respectively. Note that while in some embodiments, a small finite amount of time is 
required to acquire data over a range of wavelengths, for purposes of analysis, this 
change in time usually may be considered negligible when compared to the amount of 

1 5 time elapsing between time points used to generate a kinetic analysis. Thus, under 
these circumstances, the temporal parameter can be considered constant within a 
spectral datastack. Furthermore, the temporal parameter recorded for the data stack 
may be the actual time of data acquisition, elapsed run time at data acquisition, or any 
other parameter that relates the captured data to its time of capture. Each of these 

20 different examples of a temporal parameter is intended to be encompassed by the 
phrase "is proportional to the time of data capture." Similarly, in practice, there 
frequently is a finite range of wavelengths that are generated by even so called 
"monochromatic" light sources and/ or, selected by spectral analyzers (i.e. any device 
used to select or transmit a wavelength or a range of wavelengths). Notwithstanding 

25 this spread, a single value of wavelength can, in practice, be assigned to data collected 
under a particular condition. Choices for this value can include the mean of the range, 
the median of the range, or an energy-weighted average of the various wavelength 
components selected or transmitted by the spectral analyzer, or generated by a 
"monochromatic" light source. Each project can be identified with a single 

30 experiment, sample, or mode of data acquisition. In the latter case, Kairos imaging 
spectrophotometers can be used to acquire absorption and fluorescence spectra or 
kinetics (e.g., changes in absorbance or fluorescence over time), for the same sample. 
In enzyme kinetics studies, both absorbance and kinetic data for a particular target 
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(such as an assay disk covered with microcolonies) can be acquired. This type of data 
can be acquired by the Kairos MicroColonylmager (MCI) which is part of the "Kcat 
Technology" for enzyme screening. These datatypes represent different datastacks 
and can be stored in separate projects, which facilitates the simultaneous or combined 
5 analysis of fluorescence, absorption and/or kinetics for every pixel in the scene. 

The layout of a typical Kcat display configuration is shown in FIG. 1. In FIG. 
2, a workspace containing three different projects: absorption, fluorescence, and 
kinetics is demonstrated. The kinetics image stack has been opened to show the 
individual timepoint images. For an absorption datastack, the timepoint images would 

1 0 be replaced by images taken at specific wavelengths. 

Each project in the workspace may contain one or more analysis files, which 
contain data calculated as a function of the non-spatial dimension in the image stacks. 
Each one of these data vectors corresponds to a row in the contour. When the non- 
spatial variable is wavelength, the vector is referred to as a spectrum. When this 

1 5 variable is time, the vector is commonly referred to as velocity or a kinetic trace. 
Although the terms "spectrum" and "kinetics" are used in the description of the 
present invention, it must be noted that the methods are of general applicability to 
both forms of graphical data. Three interactive windows, consisting of a reference 
image, a contour plot and conventional plot window, are displayed. The results of a 

20 given analysis can be saved and used in subsequent data processing after the file is 

reopened. This stored information is referred to as a template, and it includes resultant 
sorts of the contour plot and pixel groupings which can later be applied to the same 
datastack or to alternative datastacks. 

The image stack category accommodates multiple image stacks. Data can be 

25 calculated from a raw image stack or pre-processed image stacks. Pre-processing can 
include a simple division or subtraction to correct for background, or more involved 
processing to correct for spectral overlap exhibited by multiple fluorescent dyes in an 
imaged sample. In kinetic experiments, it is useful to divide all images in the stack by 
the time zero image. A flat field correction may also be used. This algorithm corrects 

30 for background and restricts the grayvalue range of the processed image. This is done 
by using a background image which has previously been divided by its average 
grayvalue. With large image files, it may be beneficial to perform a prefiltering step 
which creates a smaller image stackconsisting of subsets of regions of interest in the 
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larger image. For example, this subset may comprise a collage of microcolonies 
satisfying a particular criterion. Such preprocessing has the advantage of concisely 
displaying rare positive microcolonies while significantly reducing computation time 
and file storage space requirements. In certain experiments, it is also possible to 
5 simultaneously acquire multiple datastacks representing different wavelengths for the 
same timepoint or multiple timepoints for the same wavelength. This four 
dimensional data concept is illustrated in FIG. 3. Using this concept, arithmetic 
processing of image stacks acquired under different wavelength conditions can be 
used to correct for spectral overlap exhibited by multiple fluorescent dyes in an 
1 0 imaged sample prior to an analysis. 

The image category is used to store images that are not necessarily part of the 
stack. These images can be unprocessed monochrome images acquired under special 
experimental conditions, or they can be image-processed monochrome or RGB 
pseudocolored images. Any images in the workspace can be used as a reference 
1 5 image in an analysis. 

The stack and image computation described here can be menu driven or 
incorporated into a wizard, or incorporated into firmware. 

The workspace window has been hidden in FIG. 4. Instead, multiple windows 
are shown which demonstrate various display capabilities of the software. Images can 
20 be displayed in color or monochrome and in various magnifications. Data obtained 
from a stack of images can be converted into graphical form and displayed in the plot 
window. Plots in the plot window can be shown as individual traces or as averages of 
a selection or group. This is accomplished using user- defined linewidths and colors. 
In the main application window, the main menu items, a list box, toolbar, and status 
25 bar are shown. 

The contour plot is a convenient visualization tool for concisely displaying 
three-dimensional data on a two-dimensional screen. Each thin row in the contour 
plot represents the data for a particular pixel or feature. The x-axis represents 
wavelength (for spectral data) or time (for kinetics data). Thus, each wavelength or 
30 time point is represented by a discrete column in the contour plot. Of course, this 

arrangement can be altered without departing from the scope of the invention, as by, 
e.g., rotating the contour plot through 90 degrees so that each row represents 
wavelength or time, and each column represents the data for a particular pixel or 
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feature. The intensity of the measured signal (e.g., absorbance) at a given wavelength 
or time point is indicated by a color code, whose scale is depicted at the bottom of the 
contour plot window. Black/blue color indicates no/low absorbance and red/white 
indicates high/maximum absorbance. Thus, for spectral data, the spectrum of a given 
5 pixel or feature having a single absorption maximum may be represented by a row in 
the contour plot which has a white or red segment within the column corresponding to 
the wavelength of maximum absorption. This absorption maximum is flanked by 
other colors representing progressively decreasing absorbance. Absorption, 
reflectance, or fluorescence data can be displayed for every pixel or feature in a scene. 
10 As those skilled in the art will readily appreciate, many alternatives to the above- 
described color code may be used to represent the intensity of the measured signal 
such as intensity variation (i.e., brighter or lighter regions along the contour plot row), 
and variations in any other type of visually distinguishable display variation such as 
stippling, cross-hatching patterns, or any other plotting symbol that can be related to 
1 5 signal intensity in a manner analogous to the exemplified color bar. When the pixels 
or features are sorted, the various rows are re-ordered from top to bottom in the 
contour plot window. Thus, sorting tends to create more easily recognized groupings 
of pixels or features. 
Single Pixel Versus Blob Analysis 
20 Using the GUI diagrammed above, pixels can be grouped into features by 

conventional image processing techniques, and all four of the windows within the 
GUI then act to coordinate feature-based (rather than pixel-based) information. While 
feature-based analysis can increase the signal-to-noise ratio in certain low-light 
applications, we find feature extraction to be inferior to pixel-based analysis in many 
25 applications. This is due largely to problems associated with separating neighboring 
pixels into different features that may be adjacent or overlapping. Another reason for 
basing analyses on pixels rather than features is that problems with 'edge' pixels can 
be minimized (FIG. 5). This is especially important when one is attempting to identify 
objects in a target with the highest or lowest (spectroscopic) parameter which 
30 would be otherwise averaged-out within a merged feature. 
Pedagogical Test Target 

A pedagogical, easily recognized sample is used in this section to demonstrate 
various aspects of the software. FIG. 6 shows a screen capture of the GUI, wherein a 
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plate of mini-M&M's candies (9 mm in diameter, seven colors) has been imaged in 
reflectance mode on a black background to create a 'stack' of images. The image 
window has been magnified (zoomed) so that two individual pixels falling on a 
yellow and on a green M&M can be more easily identified. The absorption plots 
5 corresponding to the reflectance spectra of the target area (approximately 25 microns 
x 25 microns) under these two pixels are shown in the plot window. As can be seen 
in the actual spectrum, the f yellow pixel' encodes data for a microscopic region within 
a single yellow M&M; its spectrum exhibits high absorbance in the blue. The 'green 
pixel' encodes data for a green M&M; its spectrum exhibits high absorbance in both 

1 0 the blue and red, with minimal absorption in the green. Both of these spectral plots 
are linked by the GUI to a yellow and a green tick mark to the left of the contour plot 
(left side of color contour plot window). Each row of the contour plot corresponds to 
a single-pixel absorption spectrum, which is color-encoded according to the rainbow- 
like color bar directly beneath. Using the row indicated by the yellow tick mark and 

1 5 referencing it with the yellow-colored conventional plot in the plot window, we first 
see (as we look from left to right in the row) white pseudocoloring of this row in the 
contour plot, which corresponds to high absorbance in the 450 - 490 nm region, then a 
rainbow of hues between 495 - 540 nm as the absorbance decreases, and finally pure 
black on the far right. 

20 The GUI is highly interactive in run-time. The computer's mouse can be used 

to point to a pixel in the image window and thereby initiate two actions: 1) the pixel 
spectrum is displayed in the plot window, and 2) a tick mark appears next to the 
associated row in the contour window. Alternatively, pointing to a row in the contour 
plot causes two analogous actions: 1) the corresponding pixel is highlighted in the 

25 image window, and 2) the associated conventional plot is updated for that particular 
pixel. Dragging the mouse vertically over the contour plot while the mouse button is 
held down selects multiple rows, whose spectra are then plotted simultaneously. 
Likewise, dragging out a box in the image window simultaneously indicates the 
corresponding spectra in both the plot and contour windows. Coordinated keyboard 

30 and mouse actions can be used to make multiple selections. Once selections are made, 
options for color display, plot linewidth, spectral averaging, and further processing are 
enabled using menu, mouse and keyboard manipulations well-known to MS Windows 
users. >; 
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Sorting and Display of Contour Plots for a Single Group 

Contour plots are very effective for visualizing and extracting useful 
information from massive amounts of spectral data. The concise display of data is 
possible because of the application of a series of sorting algorithms that group pixels 
5 with similar properties. In the case of pedagogical M&M's candies, these properties 
are due to visible light absorption. Aspects of this sorting process are shown below in 
FIG. 7. Panel A displays the desampled contour plot in a variable scale mode, 
wherein each spectrum has been stretched to full scale, so that each row displays 
absorbance intensity ranging from black to white on the absorption color-code bar. 
1 0 Panel B shows this same contour in a fixed-scale mode, wherein each spectrum has 
been normalized to the minimum and maximum absorbance values for the entire 
dataset. Both Panels A and B are unsorted and represent spectral information as it was 
initially acquired. In Panel C, we have sorted fixed-scale spectra by the maximum 
absorbance, so that the spectra with the highest optical density (represented by white 
1 5 to reddish colors) are grouped at the top of the contour plot. In Panel D, we have 
sorted the spectra within the contour plot by the sum-of-the-square-of-differences 
(SSD) using an initial yellow 'target' spectrum. A target spectrum is a spectrum that 
is specially chosen for the purpose of performing a comparison with other spectra. 
The target spectrum can be chosen from published sources (e.g., the known spectrum 
20 of chlorophyll a in ether), or alternatively, it can be selected empirically from the 
measurement of the material being imaged. The SSD equation is: 

SSD^{l A {J)-I B {j)f 



25 Where I A represents the intensity of the spectrum A at wavelength j and I B is a target 
spectrum. This calculation is performed for each row in the contour plot, and the SSD 
values are then sorted based on their distance from the target spectrum (i.e., the 
relative values resulting from operation of the of the SSD equation on the function, I A 
that relates pixel intensity to wavelength) obtained for the data represented in each 

30 row. This creates a distance measure for each spectrum. The SSD for the intensity of 
the signal can be calculated at each wavelength, as in this spectral example, or the 
SSD for the absolute values of the intensities can be calculated at each time point for 
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kinetics data. Thus, almost any discrete function can be used as a target 'graphic 5 for 
calculating a distance measure and sorting the data. As can be seen in Panel £>, 
single-pixel spectra corresponding to yellow M&M's have been isolated at the top of 
the contour plot. In Panel E, we start with an arbitrary initial target, such as the first 
5 or topmost spectrum in the contour. Similar to the processing for Panel A SSD values 
are sorted; however, this time the process is automatically reiterated with a different 
target spectrum for a total of CF-1) times, where F is equivalent to the number of 
spectra or rows in the contour. The choice of a target spectrum is discussed below. 
Target Vector Selection 

1 0 The software is capable of using a variety of different target vectors that can 

be specified by the user. There are many possible candidates for the target vector. 
This flexibility can be used in a reiterative method for rapid compression (requiring 
human intervention) or as a 'single pass' mining tool. For example, a researcher may 
be interested only in knowing whether certain spectral characteristics exist in an 

15 . image stack. In this case, a form of target set analysis can be implemented, wherein a 
previously stored reference spectrum is used as the target vector. The results of the 
first iteration of sorting will 'mine 1 spectrally similar pixels into one category in the 
contour plot. These pixels can then be color-coded on the image and removed from 
subsequent sorting. In this f sort and lock 1 procedure, average spectra and variances 

20 can be calculated and displayed. This process can be repeated using different target 
vectors until all pixels are categorized. Thus, for example, a new target spectrum can 
be selected by the user based on the appearance of the contour plot produced by the 
previous target spectrum. 

In cases in which spectral components are not known, or in which single-pixel 

25 spectra contain contributions from multiple components including instrumental or 
lighting artifacts, a randomly generated spectrum can be used. Alternatively, the 
spectrum from a random pixel or feature can be selected as a first reference. This 
latter selection is similar to the procedure described for Panels D and E in FIG. 7. 
Another way to choose the target spectrum, when spectral components are not known 

30 a priori, is to use the mean spectrum in the image stack. One can also perform a pre- 
sort, and (based on the resultant categories) use the average spectrum of a selected 
category as the target (the combined average of at least two spectra). Alternatively, 
user-defined functions can be used as target vectors. In the case of kinetics, a linear 
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function with a predefined slope or an exponential function may be beneficial. 
Although the terms spectral and kinetics are used here, it is noted that these 
discussions are equally applicable to both types of data. 
Demonstration of Target Vector Selection and Its Use with Multiple Groups 
5 Given the many possibilities for target vector selection, we demonstrate its 

application in the context of multiple groups created using the GUI. In Panel E of 
FIG. 7, numerous groupings and categories of spectra can be identified when the 
similarity sort was performed automatically with no user intervention. However, it is 
not apparent from this grouping how to back-paint the image so that background 

1 0 pixels and pixels corresponding to the seven M&M categories are clearly 

differentiated. User-input heuristics are important in guiding the particular series of 
algorithms required to sort a dataset to a level of refinement that is appropriate for 
backcoloring the image window. 

FIG. 8 displays an example each of these algorithms can be used in sequence. 

1 5 Panels A and D from FIG. 7 are repeated in FIG. 8; however, in FIG. 8 the two 

vertical color bars are also shown. In Panels B - E, yellow categories of spectra have 
been grouped by the user. This was done by a single pass SSD sort using a yellow 
target spectrum, followed by user selection of rows in the contour plot which map 
back to the image in a logical manner. Based on a real-time mapping of pixels in the 

20 image, the user can discard rows if they map to background pixels or if they populate 
other M&M candies in a random, non-continuous manner. Once the rows are 
selected, they can be delineated as a grdtip simply by selecting the menu item 
CreateGroup. When a group is created, a colored vertical bar spanning the height of 
the selected rows appears in the grouping bar to the left of the vertical black line. 

25 Individual rows within this group can still be independently selected and back-colored 
as evidenced by the small tick mark in the selection bar. Note that the contour has 
actually been divided into two groups in FIG. 8 Panel B: the yellow spectral group 
(red) and the default group (gray or not shown) which consists of the remaining rows 
in the contour. 

30 The user interface is very flexible and allows for repeated individual sorting 

within groups. Groups can also be ungrouped, combined and regrouped as necessary 
to refine a given analysis. Numerous support functions for placement of a group 
within a contour plot are also available. Individualized group sorting is demonstrated 
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in FIG. 8 Panels C and D, which are displayed in a variable scale mode. In Panel D, 
only the yellow spectral group has been sorted for maximum intensity at 475 nm. By 
selecting only the rows coixesponding to low intensity yellow spectra (identified by 
tick marks in the selection bar), one can selectively identify pixels which map to the 
5 letter M on each yellow M&M candy (also see FIG. 9). FIG. 8 Panel E shows the 
results of sequential SSD sorting using yellow, orange, pink, red, brown, green, and 
blue target spectra. Thus, a plurality of target spectra can be used to organize the 
contour plot into multiple groups for backcoloring the image. The final color-coded 
analysis is shown in FIG. 9. Pixels belonging to the background have been grouped 

1 0 and color coded as black. 
Creating an Analysis 

A flowchart outlining one embodiment of steps involved in creating 
absorbance spectra from a spectral datastack is shown in FIG. 10. A similar process 
can be used to calculate kinetic information from a time based datastack. Alternatives 

1 5 and variations of the steps enclosed in the dashed box are presented in later sections 
of this patent. Multiple examples of how to generate a Reference Image are also 
given in a later section. 

The individual steps of FIG. 10 have been incorporated as a wizard in the 
software. Analysis information includes the regions of interest (e.g. pixels or 

20 features) and the resultant sorts of a contour plot and its groupings. This information 
can be saved and later applied to the same datastack or to alternative datastacks. 
Thus, in FIG. 10 two alternative starting paths are shown: a new analysis starting with 
a Reference Image or an analysis starting with previously stored information. Each 
starting path includes the subsequent option of using feature- or pixel-based 

25 calculations. If feature-based calculations were selected, there is the additional option 
of using the average, median, highest, or lowest pixel values of each feature. Other 
parameters include whether or not to flat field the image stack and the ability to select 
a subset of images in the stack for processing. 

In this flowchart, ROI determination is based on contrast enhancement of the 

30 Reference Image. This is done automatically within the code using preset parameters 
(e.g., pixels whose values fall within the top 10% of pixel values in the image) which 
a user can override by dragging on the two sliderbars beneath the image. This allows 
one to change the respective high arid low values used to determine the ROIs. 
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Additionally, a user can paint on the image with a user defined brush size to erase 
and/or add ROIs. Similar functionality is enabled for identifying the I 0 reference in 
the Set I 0 GUI. I 0 pixel values are incorporated in the Beer-Lambert equation (Abs = 
log I 0 /I) in order to calculate absorbances. These absorbances are then displayed 
5 within a contour plot. 

Reference Image and ROI Determination 

Determination of ROIs are sometimes highly correlated to the determination 
of the reference image. Since ROIs are determined by pixel value and heuristics such 
as morphology, image processing and enhancing an image using physically relevant 
1 0 parameters is important. Frequently, useful information is already apparent from the 
reference image. Therefore, the generation of a reference image can also be 
considered a prefiltering step which minimizes the amount of data to be processed. 
For example, in screening microcolonies, one does not want to compute spectra for 
parts of the image which do not contain microcolonies. Therefore, a reference image 
15 is used to extract only those regions in the image which do correspond to 

microcolonies. In certain instances, there is no single reference image from which 
ROIs are extracted. In this case, the reference image serves as a visual aid only and 
ROIs are calculated using images and parameters entered into the software. An 
example of this situation is given below. 
20 In the simplest embodiment, a reference image is an unprocessed monochrome 

image taken at a specific timepoint under specific wavelength illumination and 
detection conditions. These images can also be background subtracted or flat-fielded 
to correct for optical and other artifacts. In all computation processes, information 
loss must be taken into consideration. For example, if division is used the resulting 
25 number may be very small, therefore, integral pixel values are first converted to 
floating point notation prior to division and rescaled before the conversion back to 
integers takes place. To facilitate display, final display values are often rescaled to an 
eight-bit range between 0 -255. 

One embodiment of a reference image combines images taken at wavelengths 
30 corresponding to known spectral parameters in the sample. For example, in 

fluorescence and absorbance, spectral images corresponding to peak maxima or 
minima can be selectively combined in an arithmetic or algebraic manner. Similarly, 
images can be ratioed using any of a combination of wavelengths. 
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Another embodiment of a reference image for prefiltering uses timecourse 
images. If the raw timecourse datastack has already been flat-fielded by the To image, 
a later timepoint image may contain kinetic data. Single timepoint images such as 
this are background corrected and they save feature or pixel information according to 
5 parameters set for maximum absorption, rather than maximum change of absorption 
over time. 

When four dimensional data is available as in an RGB image, multiple 
channel information can be combined for a particular timepoint. This embodiment of 
a reference image represents the change in absorption of a target over time. An RGB 

1 0 image is created by subtracting an image obtained at a early time-point from an image 
obtained at a later time-point. This resulting image will be black (RGB values of 
zero) wherever there is no increase in absorption and will be colored (positive RGB 
values) where increased absorption occurred over time. Another alternative is to 
derive the reference image by dividing one image by another. For example, a 24 

1 5 minute RGB image can be divided by a 2 minute RGB image after synchronous 
induction of a chromogenic reaction. This method removes fluctuations in the 
background intensity between different images. If division is used, the program 
converts the individual RGB values from integers to floating point variables during 
the operation and re-scales the values before converting them back to integers. 

20 Otherwise, the resulting RGB values will be reduced to a narrow range, and therefore 
there will also be a loss of information. 

One embodiment of the dialog boxes in FIG. 10 is presented in FIG. 11. For 
demonstration purposes, three spectral wavelengths corresponding to red, green and 
blue in an RGB image are diagrammed. These three checkboxes can be replaced with 

25 one or more listboxes to account for additional wavelengths. Input parameters 

include the timepoints, the RGB colors, and the pixel value percentage which defines 
the pixels to be selected. In this example, two timepoint images are processed and the 
resultant image displayed in the dialog box. Since none of the checkboxes are marked, 
no pixels are selected. This image is updated in real-time when the input parameters 

30 are changed. Although subtraction is shown, any number of other operations can also 
be used. 

Using the dialog box in FIGS. 1 1 and 12, selected pixels can have absorption 
in one or any combination of RGB channels. If pixels are to be selected that absorb in 
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more than one channel, the "AND" option can be checked. If pixels are to be selected 
that absorb in any of two or more channels then the "OR" option should be checked. 
When a single channel is selected, the AND and OR options are grayed. As an 
example, if the blue and red channels are chosen with the "AND" option, then the 
5 selected pixels are pseudocolored magenta. On the other hand, if the "OR' option is 
used, the selected pixels are pseudocolored blue if they were selected by the 'blue' 
criterion and pseudocolored red if they were selected by the 'red' criterion. If a 
nonexclusive OR is used, then pixels meeting both criteria are colored magenta to 
indicate the additive color combination of red plus blue. The pixels that are selected 
10 are those with absorption in the chosen channel(s) that have pixel values at or above 
the percentile entered in the dialog box. Other Boolean operators that can be used in 
the selection of pixels include NAND and XOR that can be implemented in the 
software and graphical user interface so as to provide alternative means for the user to 
prefilter pixels. All of these algorithms involving OR, AND, NAND, NOR and other 
1 5 Boolean operators can be embodied in software using Microsoft MFC/ C++ or 

alternative programming languages to create stand-alone programs. However, these 
algorithms can also be run on specialized boards that implement the equivalent 
computer code at a machine-language level as in firmware. 

The updated image with selected pixels colored magenta, is shown FIG. 12. 
20 As a visual aid for the user, a monochrome image with flashing overlay pixels can 

also be displayed. Alternatively, the user can change to a contrast-enhance view mode 
wherein the RGB image is shown and the intensity values for each color can be 
stretched to fit any range between 0 and 255. There are other alternatives for viewing 
the selected pixels. For example, having the selected pixels displayed on a black or 
25 white background instead of a monochrome image indicates exactly which pixels 
have passed through the prefilter. Another visual tool to indicate selected pixels or 
features is to trace the bordering pixels of those selected such that the bordering pixels 
alternate in color over time. 

In addition to the Boolean type processing above, multichannel information 
30 can be evaluated by color distance criteria by formulating equations that compare the 
color of all of the pixels in a series of images to preselected target values. The images 
can be any image including one of the previously described reference images above. 
Using the three channel RGB system as example, a target of 200, 10, and 30, 
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corresponding to RGB can be set. This target can be selected from the image or 
predefined based on previous experiments. A distance metric corresponding to the 
sum-of-differences between the target and each pixel's RGB value is determined and 
then compared to a specified cutoff value. If the distance cutoff were, for example, 
5 set to 30, a pixel with a value of 210, 19, 20 would be selected and a pixel with a 
value of 231, 10, 30 would be rejected. Color distance criteria also may comprise 
alternative equations such as the sum-of-the-square-of-the differences, i.e., an actual 
color distance in RGB color space. 
Data Compression 

0 While making a transition from feature to pixel processing, we saw an 

opportunity to contribute to hyperspectral database management after realizing that 
contour plots can facilitate new data compression methods. Hyperspectral information 
can be significantly compressed by using novel algorithms which eliminate data loss 
when used in the context of a client/server protocol. Based on the initial rapid 

5 preview of highly compressed data, a subsequent request for more specific 

information can be sent. This integrated approach to hyperspectral data management 
is needed in many fields where spectral datacubes are beginning to emerge as new 
instrumentation is developed. These fields include remote sensing and telemedicine 
where data is shared and transmitted to individual researchers over communication 

10 lines. 

Contour plots are readily linked to data compression and take advantage of 
spectral heuristics, unlike common graphics image compression methods, which do not 
Because image stacks are formed from grayscale 2D images, well known formats such 
JPEG and GIF will either be poor at spectral compression or generate loss. These data 
25 compression methods do not take into account the relationships of information in an 
added dimension which can be used to enable the compression in the special cases 
discussed here. 

Using the M&M sorting examples above, the compressed image stack can be 
reduced into data elements consisting of one color-coded image and the spectra and 
30 variances of each category. A stack of N images is essentially reduced into one 
image, a desampled contour plot, and spectral summary information. The 
compression factor is approximately equal to the number of images in the stack, N. 



WO 01/54067 PCT/US0 1/02 180 

23 

Compression Factor ~ N Eqn. 2 



In cases where spectral categories are not well defined, as in the red and pink 
M&M's, compressed data can be supplied with the red and pink categories 
5 hypothetically grouped as one. Based on the spectra and variances also supplied, a 
spectral envelope and variance of these pixels can be generated and displayed in the 
conventional plot window of the GUI as part of the compressed data. In a mock 
client/server scenario, this transmitted information showing the large variance at 
longer wavelengths is indicative of a distribution of spectra which can be separated 

1 0 into more than one category. Such an initial preview of compressed information 

would prompt the client to request more detailed information, which can be isolated to 
a smaller and more specific subset of pixels. 

The GUI platform described above is amenable to a 'sort and lock procedure, 
which can be used to reduce computation time and facilitate compression. Multiple steps 

15 in a spectral analysis process can be performed to produce a series of contour plots, each 
one resulting in the identification of one or more spectral categories. Once these pixels 
are defined, they can be locked out and excluded from subsequent processing, thereby 
decreasing the number of pixels to process in the subsequent step. This 'sort and lock 1 
procedure is presented as an alternative to an MNF transformation and end-member 

20 analyses (Green, A. A., Berman, M., Switzer, P, & Craig, M. D. (1988) A transformation 
for ordering multispectral data in terms of image quality with implications for noise 
removal: IEEE Transactions on Geoscience and Remote Sensing, v. 26, no. 1, p. 65-74.; 
Boardman J. W., & Kruse, F. A. (1994) Automated spectral analysis: A geologic 
example using AVIRIS data, north Grapevine Mountains, Nevada. In: Proceedings, Tenth 

25 Thematic Conference on Geologic Remote Sensing, Environmental Research Institute of 
Michigan, Ann Arbor, MI, Vol. 1, pp. 407-418) which also seek to reduce the amount of 
data processed. A useful method of the present invention is to sort the data in the contour 
plot such that they can be compressed to representative spectra. 
Examples of Sorting Strategies 

30 Here, we demonstrate a series of steps which can be used to screen a bacterial 

library for enzyme variants with the fastest kinetics as well as the highest specificity 
for a particular reaction. For example, combinatorial cassette mutagenesis has been 
used to generate a recombinant library of over 10 million variants of Agrobacteriam 
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beta-glucosidase (Abg), a sugar-cleaving enzyme. Since this enzyme has a broad 
substrate specificity, different substrates such as glucoside and galactoside, can be 
tagged with different chromogenic reporters. Experiments were conducted using two 
indolyl derivatives; Red-gal and X-glu. Galactoside and glucoside specificities were 
5 identified by absorbance at 540 nm and 61 5 nm corresponding to the lambda max of the 
respective indigo products formed from each derivative. Thus, the 'bluest' pixels 
would correspond to variants having the highest substrate specificity for glucoside 
and the 'reddest 5 pixels would correspond to variants having the highest substrate 
specificity for galactoside. In the following examples, a time-based image stack was 

1 0 first acquired from T 0 to T n corresponding to time 0 to time 2700 seconds at a 

wavelength of lambda =610 nm. Following this, a spectral stack was acquired over 
the wavelength range lambda to lambda m corresponding to 500 nm to 700 nm. 
These two datastacks were stored in separate projects called Absorbance and 
Timecourse respectively. These examples illustrate how the images can be analyzed 

15 so that the pixels ultimately displayed are restricted by at least two independent 

criteria. A generalized flowchart of steps including those described in Example 1 A 
are shown in FIG. 13. 
EXAMPLE 1 

In this example, an analysis was first performed using the spectral data 
20 obtained at the end of a 45 minute kinetic run to select pixels with the greatest 610 nm 
: 540 nm absorbance ratio. After data acquisition, the following steps were followed: 

1. The 610 nm image from the Absorbance project was flat-fielded with 
the T 0 image from the Timecourse project in order to generate a reference image. 

2. Using this reference image, a pixel-based spectral analysis is created. 
25 The software wizard automatically identifies pixels with grayvalues in the lowest 

tenth percentile. These pixels correspond to the pixels with the highest absorbances at 
610 nm. As previously discussed, the user can override these automatic selections by 
painting on the image or redefining the high and low grayvalue range for ROIs. 

3. A microcolony-free region on the edge of the membrane disk is 

30 selected for I 0 . These values are input into the Beer-Lambert equation to calculate 

pixel absorbances and the intensities are then displayed in a contour plot as a function 
of wavelength. 
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4. The spectral analysis is sorted by the ratio of intensities at 540 nm and 
610 nm, corresponding to the larabda max of the indigo products resulting from 
hydrolysis of Red-gal and X-glu, respectively. If the resultant sort is displayed in 
descending order, the 'reddest' pixels appear at the top and the 'bluest ' pixels appear 

5 at the bottom of the contour plot. 

5. The bluest pixels are marked and categorized into a new group which 
is identified by the red grouping bar in FIG. 14. Pixels in this select group have been 
backcolored onto the image in blue. Their respective absorbance plots are also shown 
in the plot window of FIG. 14. 

10 6. Using the stored analysis of step 5, a new pixel-based kinetic analysis 

is generated in the Timecourse project. By applying this existing analysis, all of the 
pixels which were determined in the spectral analysis can be analyzed in this 
timecourse analysis. Furthermore, grouping information is maintained so that 
selective sorting can be performed only on pixels previously associated with the 

1 5 highest substrate specificity for glucoside (e.g., bluest ). 

7. A microcolony-free region on the edge of the membrane disk is 
selected for I 0 and parameters are set so that the images are flat-fielded with the T 0 
image prior to the Beer-Lambert calculation. These intensities are then displayed in a 
contour plot as a function of time. 

20 Since an existing analysis was used, grouping information is retained and also 
displayed. 

8. The previously selected group of 'bluest' pixels is moved to the top of 
the contour and sorted by the maximum value at T = 600 seconds, a timepoint where 
the rate of change in intensity is linear. In order to compare relative intensities, the 

25 contour plot is displayed in a fixed scale mode. Pixels corresponding to the subset of 
'bluest pixels' displaying tlie fastest product formation are sorted to the top. In FIG. 
15, these pixels have been backcolored onto the image in blue and their respective 
kinetic plots are also shown. 
EXAMPLE 2 

30 In this second sorting example, an analysis was first performed using the 

timecourse data obtained during a 45 minute kinetic run to select pixels meeting 
specific temporal criteria. In other examples, this kinetic run can be longer or shorter. 
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In this case, the temporal criterion is the fastest absorbance increase at 610 nm. The 
following steps were followed: 

1 . The T = 600 second image is flat-fielded with the T 0 image from the 
Timecourse project in order to generate a reference image. 
5 2. Using this reference image, a pixel-based kinetic analysis is created. 

The software wizard automatically identifies pixels with grayvalues in the lowest 
tenth percentile, corresponding to high absorbance at 610 nm and significant Abg- 
catalyzed product formation. As previously discussed, the user can override these 
automatic selections by painting on the image or redefining the high and low 

1 0 grayvalue range for ROIs. 

3. A microcolony-free region on the edge of the membrane disk is 
selected for I 0 . After the images in the stack are flatfi elded with the To image, pixel 
absorbances are calculated according to the Beer-Lambert equation and then 
displayed in a contour plot as a function of time. 

15 4. In order to compare relative intensities, the contour plot is viewed in a 

fixed scale mode and then sorted by the maximum value at T = 600 seconds; a 
timepoint where the rate of change in intensity is linear. Depending on the number of 
pixels, it may be necessary to display the contour in a desampled mode as previously 
described. 

20 5. The fastest pixels are marked and categorized into a new group which 

is identified by the red grouping bar in FIG. 16. Pixels in this select group have been 
backcolored onto the image in blue. Their respective kinetic plots are also shown in 
the plot window of FIG. 16. 

6. Using the stored analysis of step 5, a new pixel-based absorbance 

25 analysis is generated in the Absorbance project. By applying this existing analysis, all 
of the pixels which were selected in the timecourse analysis can be processed in this 
spectral analysis. Furthermore, grouping information in the image stack is 
maintained so that selective sorting is performed only on pixels previously associated 
with the fastest rate of Abg-catalyzed product formation, as determined at time T = 

30 600 seconds. 

7. A microcolony-free region on the edge of the membrane disk is 
selected for Io and the absorbances are calculated. These intensities are then displayed 
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in the contour plot as a function of wavelength. Since an existing analysis was used, 
grouping information is retained and also displayed. 

8. The previously selected group of 'fastest' pixels is now sorted by the 
ratio of intensities at 610 nm and 540 nm corresponding to the lambda max of the indigo 
5 products resulting from X-glu and Red-gal, respectively. The 'bluest' pixels now 

appear at the top and the 'reddest 6 pixels appear at the bottom of the sort. In FIG. 17, 
a subset of 'fastest pixels' displaying the bluest product formation has been 
automatically indicated by backcoloring the reduced subset of pixels in the image. 
EXAMPLE 3 

10 In a third sorting example, spectral data obtained at the end of a kinetic run (or 

during the run) is used to determine ROIs meeting a specific spectral criteria without 
performing a complete contour plot based spectral analysis. This is done by 
generating a reference image from absorbance images as previously described. Using 
the Abg experiment as an example, the 610 nm image can be divided by the 540 nm 

15 image and the pixels with the lowest grayvalues would correspond to the 'bluest' 
pixels. If a satisfactory pixel cutoff value has been previously determined, one can 
use this cutoff value to select ROIs without performing the entire spectral analysis and 
sorting described in steps 1-5 of EXAMPLE 1 above. A single reference image 
based on spectral data is generated and this image is used for the kinetic analysis as 

20 listed in steps 7 - 8. 
EXAMPLE 4 

In a fourth sorting example, timecourse data is used to determine ROIs 
meeting a specific temporal criteria without performing a complete contour plot 
based kinetic analysis. 

25 This is done by generating a reference image from timecourse images as 

previously described. Using the Abg experiment as an example, the T = 600 second 
image can be flatfielded by the T 0 image. This would be meaningful only if it is 
separately determined that the timepoint selected, in this case T = 600, represents a 
linear rate of change of product formation with time. 

30 In this case, the pixels with the lowest grayvalues would then correspond to the 
'fastest' pixels. 

In this example, the entire timecourse analysis and sorting described in steps 1 
- 5 of EXAMPLE 2 above, may not be necessary. A single reference image based on 
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timecourse data is generated and this image is used for the spectral analysis as listed 
in steps 7-8. 
EXAMPLE 5 

Hematoxylin and eosin (H&E) stains are performed on almost all biopsied 
5 tissues before any other special stain or immunochemical analysis is considered. As a 
result, there are approximately 10,000 H&E stained thin sections analyzed per day in 
the United States. However, the staining process is difficult to control, and 
information obtained from a stained thin section is often based on very subtle color 
differences. Standardization and visual enhancement of such differences can be 

1 0 achieved by employing imaging spectroscopy, and this capability could benefit the 
entire histology community. Here we demonstrate how several of the sorting 
algorithms of the present invention can be used to analyze datastacks acquired by 
imaging a slide of H&E stained tissue. 

FIG. 18 shows the single-pixel data obtained by acquiring an image stack of 

1 5 H&E stained tissue at a number of wavelengths. Eosin, which absorbs maximally 
between 500 nm and 550 nm, is used as a contrast stain. Hematoxylin, which is 
known to preferentially stain cell nuclei, contributes an additional small absorption 
shoulder at about 610 nm to some of the pixels. Pixels can therefore be sorted based 
the ratio of absorbance at 540 nm to the absorbance at 610 nm in order to 

20 automatically classify cytological features. This sorting process can be enhanced by 
using grouping bars. 

The process begins by sorting the single-pixel spectra based on maximum 
absorbance value. This initial sort tends to move all of the pixels representing heavy 
to moderately stained regions in the image to the top of the contour plot, whereas 

25 unstained or poorly stained regions in the image are sorted to the bottom of the 

contour plot. By clicking and dragging a grouping bar (dark green) next to the low- 
absorbance pixels in the contour plot, these pixels can be locked out of the subsequent 
sort. A second sort is then performed on the remaining high-absorbance pixels based 
on the ratio of absorbance at 540 nm to the absorbance at 610 nm. Pixels having a 

30 high ratio (i.e., regions stained primarily with eosin) are thereby sorted to the top and 
can be grouped for further processing using the violet-colored grouping bar. Pixels 
having a lower ratio due to the presence of a shoulder at 610 nm (i.e., regions that 
have been stained with hematoxylin) are sorted beneath the high-ratio group, and fall 
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into the middle of the contour plot. This small group of pixels can also be grouped for 
further processing using the light blue grouping bar. 

Each of the three classes of pixels can also be selected by clicking and 
dragging a selection/mapping bar next to the appropriate section of the contour plot. 
5 The average spectrum for a selected group of pixels is thereby displayed in the plot 
window and the pixels are pseudocolored in the image window. In this example, the 
pixels representing low-staining or unstained regions of the tissue (the bottom third of 
the contour plot) have been selected by clicking and dragging a light green selection 
bar next to this portion of the plot. The average spectrum of these pixels is displayed 

10 in light green in the plot window, and the corresponding pixels are backcolored light 
green in the image window. The small number of pixels in the middle of the contour 
plot that represent tissue regions stained with hematoxylin have been selected with a 
dark blue selection/mapping bar. Their average spectrum (which has a shoulder at 
about 610 nm) is shown in the plot window, and the corresponding pixels are 

1 5 backcolored dark blue in the image window. Note that the backcolored areas for 

these pixels correspond predominantly to the cell nuclei. Finally, the pixels at the top 
of the contour plot (with absorbance primarily at 540 nm) have been selected with a 
red selection/mapping bar. Their average spectrum is shown in red in the plot 
window, and the corresponding pixels have been backcolored red in the image 

20 window. These pixels highlight areas in the tissue that have been stained primarily 
with eosin. 

FIG. 19 shows the effects of various sorting algorithms on contour plot data 
for H&E samples. FIG. 19 Panels shows an unsorted contour plot in full scale mode. 
Several pixels have been selected from the middle of the contour plot (at the tic 

25 marks), and their spectra are displayed in the lower portion of the panel. Panel B 
shows the results of sorting spectra in the variable scale mode based on the ratio of 
optical absorbance at 540 nm compared to the absorbance at 610 nm. Pixels having 
spectra with a single absorption maximum at about 540 nm have been sorted to the 
top of the contour plot. Several of these pixels have been selected using the red 

30 selection/mapping bar, and their average spectrum is displayed in red in the lower 
portion of Panel B. Pixels having spectra with an absorption shoulder at about 610 
nm have been sorted to the bottom of the contour plot. Several of these pixels have 
been selected using the blue selection/mapping bar, and their average spectrum is 



WO 01/54067 



30 



PCT/LJSO 1/02 ISO 



displayed in blue in the lower portion of Panel B. Panel C shows the results of a ratio 
sort of first-derivative spectra. Eosin-specific pixels having an inflection point at 
approximately 530 nm have been sorted to the top of the contour plot. Several of 
these pixels have been selected using the red selection/mapping bar, and their average 
5 first-derivative spectrum is displayed in red in the lower portion of Panel C 

hematoxylin-containing pixels have been sorted to the bottom of the contour plot. 
Several of these pixels have been selected using the blue selection/mapping bar, and 
their average first-derivative spectrum is displayed in blue in the lower portion of 
Panel C. First-derivative analysis is useful for identifying small spectral shifts in 
10 large datasets. 
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What is claimed: 

1 1 . A method for analyzing digital image data, comprising: 
2 

3 (a) loading into a computer memory a plurality of datastacks wherein each 

4 datastack comprises pixel intensity data for a plurality of images, the pixel intensity 

5 data expressed as a function of: (i) pixel position, (ii) a first non-positional variable, 

6 and (iii) a second non-positional variable, wherein within a datastack, the value of the 

7 first non-positional variable is not constant, and the value of the second non-positional 

8 variable is constant, and wherein between datastacks, the value of the second non- 

9 positional variable differs; 
10 

1 1 (b) generating for a plurality of pixels within a first datastack, a plurality 

12 of first functions that relate pixel intensity to the first non-positional variable; 
13 

1 4 (c) sorting the pixels within the first stack according to a first value 

1 5 obtained by applying a mathematical operation to the first functions generated for the 

1 6 plurality of pixels; 
17 

1 8 (d) selecting a first set of sorted pixels; 

19 

20 (e) generating for a plurality of pixels within the first set, a plurality of 

21 second functions that relate pixel intensity to the second non-positional variable; and 
22 

23 (f) sorting the pixels within the first set according to a second value 

24 obtained by applying a second mathematical operation to the second functions 

25 generated for the plurality of pixels within the first set. 

1 2. The method of claim 1, wherein the first and the second non-positional 

2 variables are selected from the group consisting of a spectral parameter, a temporal 

3 parameter, a thermal parameter, a pH parameter, a chemical activity parameter, a 

4 pressure parameter, a partial pressure parameter, and an ionic strength parameter. 

1 3. The method of claim 1, wherein the first non-positional variable is a spectral 

2 parameter and the second non-positional variable is a temporal parameter. 

1 4. The method of claim 3, wherein the spectral parameter is selected from the 

2 group consisting of a wavelength of light illuminating a sample from which an image 

3 is captured, and a wavelength of light selected by a spectral analyzer interposed 

4 between a sample and an image capture device. 

1 5. The method of claim 3, wherein the spectral analyzer is selected from the 

2 group consisting of a filter, a plurality of filters, a monochromator, and an 

3 interferometer. 



1 
2 



6. The method of claim 3, wherein the temporal parameter is proportional to the 
time of data capture. 



WO 01/54067 



PCT/US01/02IKO 



32 

1 7. The method of claim 1, wherein the first non-positional variable is a temporal 

2 parameter and the second non-positional variable is a spectral parameter. 

1 8. The method of claim 7, wherein the spectral parameter is selected from the 

2 group consisting of a wavelength of light illuminating a sample from which an image 

3 is captured, and a wavelength of light selected by a spectral analyzer interposed 

4 between a sample and an image capture device. 

1 9. The method of claim 7, wherein the spectral analyzer is selected from the 

2 group consisting of a filter, a plurality of filters, and an interferometer. 

1 10. The method of claim 7, wherein the temporal parameter is proportional to the 

2 time of data capture. 

1 11. The method of claim 1, wherein the first set selected in step (d) is a subset of 

2 the pixels sorted in step (c). 

1 12. The method of claim 1 , wherein said first mathematical operation comprises a 

2 step selected from the group consisting of calculating the first function maximum, 

3 calculating the first function minimum, calculating a ratio of values within the first 

4 function, calculating a value of the first function at a predetermined time point, and 

5 calculating a distance between the first function and a target function. 

1 13. The method of claim 12, wherein the target function is a target spectrum. 

1 14. The method of claim 13, wherein the target spectrum is calculated from data 

2 within one or more datastacks. 

1 15. The method of claim 13, wherein the target spectrum is the spectrum of a pixel 

2 selected by a user. 

1 16. The method of claim 1, wherein said second mathematical operation 

2 comprises a step selected from the group consisting of calculating the second function 

3 maximum, calculating the second function minimum, calculating a ratio of values 

4 within the second function, calculating a value of the second function at a 

5 predetermined time point, and calculating a distance between the second function and 

6 a target function. 



1 17. The method of claim 16, wherein the target function is a target spectrum. 

1 18. The method of claim 1 7, wherein the target spectrum is calculated from data 

2 within one or more datastacks. 

1 19. The method of claim 17, wherein the target spectrum is the spectrum of a pixel 

2 selected by a user. 

1 20. The method of claim 1, wherein the results of sorting step (c) are displayed on 

2 a computer monitor prior to said selecting step (d). 
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1 21 . The method of claim 1, wherein the results of sorting step (f) are displayed on 

2 a computer monitor. 

1 22. The method of claim 1, wherein said datastacks comprise data selected from 

2 the group consisting of temporal changes in fluorescence signals, temporal changes in 

3 FRET signals, temporal changes in reflectance signals, temporal changes in 

4 absorption signals, fluorescence spectra, FRET spectra, reflectance spectra and 

5 absorption spectra. 

1 23. A graphical user interface for display and analysis of digital image data 

2 comprising: 

3 (a) a reference window for displaying a reference image comprising 

4 pixels; 
5 

6 (b) a contour plot window for indicating pixel location along a first 

7 dimension, indicating a non-positional variable along a second dimension, and 

8 indicating pixel intensity by a variable signal appearing along the second dimension, 

9 said contour plot window further comprising (i) a grouping bar for grouping together 

1 0 pixels for analysis; and (ii) a selection bar for selecting pixels that are thereby 

1 1 indicated in the reference window and plotted in the plot window; 
12 

13 (c) a plot window for displaying a plot of pixel intensity as a function of 

1 4 the non-positional variable. 

1 24. The graphical user interface of claim 23, wherein the non-positional variable is 

2 selected from the group consisting of a spectral parameter, a temporal parameter, a 

3 thermal parameter, a pH parameter, a chemical activity parameter, a pressure 

4 parameter, a partial pressure parameter, and an ionic strength parameter. 

1 25. The graphical user interface of claim 24, wherein the non-positional variable is 

2 a spectral parameter. 

1 26. The graphical user interface of claim 24, wherein the non-positional variable is 

2 a temporal parameter. 

1 27. The graphical user interface of claim 23, wherein said indicating in the 

2 reference window comprises a step selected from the group consisting of 

3 pseudocoloring the pixel, creating a border around the pixel, and alternating between 

4 different levels of pixel intensities. 

1 28. The graphical user interface of claim 23, wherein said variable signal is 

2 selected from the group consisting of color variation, intensity variation, stipple 

3 variation, and cross-hatch variation. 
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